Knowledge mobilization activities to support decision-making by youth, parents, and adults using a systematic and living map of evidence and recommendations on COVID-19: protocol for three randomized controlled trials and qualitative user-experience studies

Introduction The COVID-19 pandemic underlined that guidelines and recommendations must be made more accessible and more understandable to the general public to improve health outcomes. The objective of this study is to evaluate, quantify, and compare the public’s understanding, usability, satisfaction, intention to implement, and preference for different ways of presenting COVID-19 health recommendations derived from the COVID-19 Living Map of Recommendations and Gateway to Contextualization (RecMap). Methods and analysis This is a protocol for a multi-method study. Through an online survey, we will conduct pragmatic allocation-concealed, blinded superiority randomized controlled trials (RCTs) in three populations to test alternative formats of presenting health recommendations: adults, parents, and youth, with at least 240 participants in each population. Prior to initiating the RCT, our interventions will have been refined with relevant stakeholder input. The intervention arm will receive a plain language recommendation (PLR) format while the control arm will receive the corresponding original recommendation format as originally published by the guideline organizations (standard language version). Our primary outcome is understanding, and our secondary outcomes are accessibility and usability, satisfaction, intended behavior, and preference for the recommendation formats. Each population’s results will be analyzed separately. However, we are planning a meta-analysis of the results across populations. At the end of each survey, participants will be invited to participate in an optional one-on-one, virtual semi-structured interview to explore their user experience. All interviews will be transcribed and analyzed using the principles of thematic analysis and a hybrid inductive and deductive approach. Ethics and dissemination Through Clinical Trials Ontario, the Hamilton Integrated Research Ethics Board has reviewed and approved this protocol (Project ID: 3856). The University of Alberta has approved the parent portion of the trial (Project ID:00114894). Findings from this study will be disseminated through open-access publications in peer-reviewed journals and using social media. Trial registration Clinicaltrials.gov NCT05358990. Registered on May 3, 2022 Supplementary Information The online version contains supplementary material available at 10.1186/s13063-023-07067-9.


Introduction
As health practitioners, policymakers, and the public are inundated with information, misinformation, and health recommendations about COVID-19, we have built a unique repository of trustworthy COVID-19 recommendations with funding from the Canadian Institutes of Health Research (CIHR) (covid 19. recmap. org). The overarching goal of this COVID- 19 RecMap effort is to identify all COVID-19 guidelines, assess their credibility and trustworthiness, and make their recommendations available and understandable to various stakeholder groups [1].
To enhance the understanding of health recommendations by the public, we developed a multi-stakeholder process to draft, edit, and publish plain language recommendations (PLRs) [2]. Until now, plain language versions of recommendations have been poorly explored, and our work on the RecMap indicates an absence of this critical knowledge mobilization tool for the general public, including youth, adults, and parents [3]. Existing trials on tailored recommendations presentations have been small and predated recent guidance on how to present guideline recommendations, targeted health professionals, or did not target the general public [4][5][6][7]. For example, a small trial with 84 mental health service users suggested an improved intention to follow recommendations when written in plain language [6]. Trial data also exists on specific aspects related to health information, like comparing alternative ways of presenting numerical or statistical information and formats of information sharing [4,[8][9][10]. However, trustworthy and comprehensive PLRs may need to include the clinical or public health background related to the topic, conflicts of interest, available research evidence, judgments that are made, the rationale for a recommendation, the actual recommendation, and implementation considerations. As addressed in Evidence to Decision (EtD) frameworks, these facets are deemed essential to inform decisionmaking and are used widely by various organizations [11][12][13][14].
A trial is necessary to empirically show that PLRs convey the intended messages from guidelines to broader populations and not only to the selected user groups that usually participate in ours and others' research. Understanding and interpreting recommendations correctly are the essential prerequisites for the general public to become effective self-managers of their health and to ultimately optimize behaviors and related health outcomes [15]. Creating health information for the public that is accessible, reliable, and understandable is critical to scale science and evidence-based guidelines, which is especially important during the COVID-19 pandemic. We plan to undertake this trial, given the paucity of empirical evidence to guide our team and guideline developers as to how to best present PLRs and EtD facets to the public, which are critical to support their informed decisionmaking. This protocol describes the trials and details of the three leading trial sites.

Objectives
The objective of this study is to compare end-users' (youth, parents, and adult populations) understanding, accessibility and usability, satisfaction, intention to implement, and preference for COVID-19 recommendations when presented as plain language recommendation summaries (PLRs) versus standard language versions (SLVs). We hypothesize that there will be a difference in understanding of information between the two formats. We also aim to understand the reasons for their choices through our linked qualitative research.

Methods
We will follow a multi-methods approach: randomized controlled trials (RCTs) with qualitative interviews among a subset of participants. The trials are pragmatic allocation-concealed, blinded superiority RCTs with at least 240 participants in each population (parents, adult, and youth). The trials were preceded by preparatory engagement work with Canadian youth advisors, a Canadian parent advisory group, and an international adult Cochrane Consumer group, herein referred to as stakeholder groups. The youth site's preparatory engagement work was co-developed and implemented by two youth partners (MP and JR) in collaboration with LS, to ensure meaningful engagement efforts were embedded into the research process. We gathered input from stakeholders on COVID-19 priority topics and PLR content and format. Input from stakeholders helped us refine the PLR format to be evaluated with each population in the trial.
We will conduct qualitative interviews with a subset of the participants who completed the trials to help contextualize the digital user experience and understand the results of the trial.
Any amendments to this protocol will be made available via medRxiv.

Randomized controlled trials
This protocol has been prepared in accordance with the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) reporting guideline (Supplementary Material 1) [16].

Design and setting
There are three online survey links, one for each respective population. Each link contains eligibility questions, the consent form, and study information for youth, parents, or adults. Participants will access the online survey corresponding to their respective population to enter the trials and provide implied consent by participating in the survey. Using the concealed allocation code of the intervention platform, SurveyMonkey (surve ymonk ey. com), participants will select their geographic region. Participants from each geographic region will be randomized to one of two recommendation topics, and then they will be randomized to a PLR format (intervention) or a SLV format (control) of the recommendation (Fig. 1). The SLV is the original PDF guideline developed and published by an organization, such as the World Health Organization, from where the PLR information was extracted. Participants will complete an online survey consisting of Likertscale questions, multiple-choice questions, open-ended questions, and demographic information. The survey is anonymous.
Following completion of the online survey, we will invite participants to take part in an optional qualitative interview that will allow them to reflect on their preference for the PLR or SLV format and help us understand the results of the quantitative survey. Confidentiality will be maintained by storing names and emails on a password-protected device and secure network to which only authorized study personnel have access to. If participants decide to provide their name and email address to participate in the interview, receive the study results, or enter the draw (for a chance to win one out of five $50 CAD gift cards for the adult and parents' sites, one out of ten $25 CAD gift cards for the youth site), this personal information will not be linked to their survey answers. This is not a clinical trial; thus, there are no foreseeable harms or health risks from participating. However, there is a possibility that participants may experience distress once they better understand the uncertainty in the underlying evidence, even if the recommendation itself is judged trustworthy. We provide the email address and contact information of the principal investigator and research coordinator. Participants can contact us to report any discomfort or seek further information.

Participants
Each of the three leading sites will recruit participants from one of the three population groups: adults, parents, or youth. SickKids Research Institute will lead the recruitment of youth, Western University will lead the recruitment of adults, and The University of Alberta will lead the recruitment of parents. Global recruitment efforts will take place via social media. Coordination and oversight of the trials rest with the research team at McMaster University (RC and HJS). Our study is also open for other sites who are interested in participating as active recruitment sites for one or more populations. For example, St. Elizabeth University in Slovakia is joining as an active recruitment site for all three populations, with a commitment to recruit up to 60 participants.

Selection criteria
In this study, adults are defined as individuals selfreported to be 21 years of age or older, youth are defined as individuals self-reported to be between the ages of 15 and 24 years [17], and parents are defined as individuals self-reported to be 18 years or above and are a parent or legal guardian of a child under 18 years old. Participants can be from any country. For all three populations, participants will need to be able to read and understand English. We expect variation in the English language proficiency and health literacy level among participants. Therefore, when collecting demographic information, questions related to language proficiency and familiarity with COVID-19 health information will be included to understand the participant's level of health literacy. We may adjust for these variables in our analysis if needed, although we expect them to be reasonably balanced between the SLV and PLR groups due to randomization.

Recruitment
Each leading site will run recruitment for the different population groups separately yet with similar and overlapping strategies. We will recruit English-speaking (mother tongue or fluent) participants globally through our experienced RecMap investigator team (https:// covid 19. recmap. org/ about), the Cochrane Consumer Network and various Cochrane networks, guideline co-authors who interact with youth (including the international Young Persons' Advisory Groups), parents, and adult citizens. We plan to run public recruitment campaigns through identified youth, adult, and parent organizations' social media platforms (e.g., organizations' accounts on Twitter, Instagram, and Facebook) and newsletters. Each leading site will be responsible for monitoring its own recruitment, though all investigators will support overall recruitment for the trials (e.g., if recruitment falls behind for one of the populations, other sites will help support recruitment for that group). Surveys will be administered through SurveyMonkey, where participants are screened for eligibility prior to allocation. Previous studies suggest that the target recruitment rates for such trials are achievable in less than 3 months, and we will recruit at least 240 participants per group [18][19][20].
The average survey completion time is estimated to be 15-25 min, based on stakeholder feedback and pilot testing. As suggested by our stakeholders in the preparation of this trial, we will be inviting participants who complete the survey to enter a draw for a chance to win one of five $50 CAD gift cards for the adult and parents' populations. The youth group will invite youth participants to enter a draw for one of ten gift cards valued at $25 CAD. This is offered as a gesture and appreciation for completing the survey.

Intervention and comparison
In these trials, the intervention is the PLR, and the comparison is the SLV. Figure 2 shows an example of the SLV format of a recommendation that will be used with the adult and youth participants, and Fig. 3 shows an example of a PLR format that will be presented to adult participants. Since this is a non-clinical trial, receiving either the intervention or the control will not require alteration to any part of the participant's life. Additionally, we do not have special criteria for discontinuing or modifying the allocated intervention.
Through our pre-trial engagement work, we gathered input from youth (via online workshops) and from adult stakeholders (via online meetings) on high-priority COVID-19 topics and on PLR content and format. Input from youth and adult stakeholders allowed us to choose two recommendations for the trials and refine the PLR format. For the parents' population, the PLRs were refined from collected feedback during online meetings with parent groups.
All PLRs for the trials were developed from the same template and refined using stakeholder feedback. The youth and parent PLRs' design and formatting are the same, as these groups gave similar feedback, while the adult PLRs are slightly different.
In the trials, we will ask participants to read one format of a recommendation (PLR or SLV) to which they have been randomly allocated and then answer the online survey that takes an average of 15-25 min to complete.

Primary outcome
Understanding We define understanding as the comprehension of key guideline content (e.g., year of publication, intent of recommendation, direction of recommendation, etc.). This outcome will be measured using seven multiple-choice questions about key concepts in the recommendation with four to six response options for each question and only one correct answer (total minimum score of 0, maximum score of 7). Based on previous work, we aim to detect a difference of 10% in understanding between groups which we consider an important difference (an average difference of 1 correct answer) [18,19,[21][22][23].

Secondary outcomes
Accessibility and usability We define accessibility and usability as the ability to find/access and use the presented information. This outcome considers the three following domains: (1) how easy it was to find information, (2) how easy it was to understand the information (perception), and (3) whether the information was presented in a way that could be helpful for making an informed health decision. Participants will be asked to indicate the degree of agreement with six statements, measured using the 7-point Likert scale (1 being strongly disagree, 4 being neutral, and 7 being strongly agree). We will also include 1 open-ended question to get additional input on accessibility and usability. The outcome "overall accessibility/usability of information" will be measured using the average of the 7-point Likert-type scale questions.

Satisfaction
We define satisfaction as participants' impression of the recommendation's presentation. We will ask participants about their level of satisfaction with different features of the format (e.g., order of information, length of document, etc.) [24]. Participants will be asked to indicate the degree of satisfaction with three questions measured using the original 7-point Likert-type scale (1 being very dissatisfied, 4 being neutral, and 7 being very satisfied). We will also include two open-ended questions to get their input on what they liked and disliked in the format.
Intended behavior We define intended behavior as participants' intention in adopting and following the shared recommendation. We will ask participants if they have already followed the recommendation and have them respond with yes, no, unsure, prefer not to answer, or not applicable. Subsequently, we will ask how likely it is that they will follow the recommendation (if they have not already) and share the recommendation with others, through two questions measured using a 7-point Likerttype scale (1 being very unlikely, 4 being neutral, and 7 being very likely). Preference We define preference as a greater liking of one format over the other (PLR or SLV). After participants complete the tasks in the group they are randomized to, they will be asked to compare the PLR and the SLV formats. Participants will review the alternative format and indicate their preference for one of the two formats using a 7-point Likert-type scale (1 being strongly preferring the SLV, 4 being the same preference for both, and 7 being strongly preferring PLR). Participants will also have the chance to explain their choice in a text box.

Stratification
We will stratify our participants by region: Africa, Americas, Eastern Mediterranean, Europe, South-East Asia, and Western Pacific. We have not set a target sample per strata. The rationale behind stratification is to ensure an equal chance for participants from the same region to be randomized to either the intervention or control (as opposed to having all participants from Europe, for example, being randomized to the PLR format).

Randomization
After stratification, participants will be randomly assigned in a 1:1 ratio to recommendation topic 1 or 2, then will be randomized again in a 1:1 ratio to the PLR or the SLV arms (see Fig. 1). Randomization will ensure that recommendation formats will be equally distributed across participants to get balanced judgments on outcomes.

Allocation concealment
The allocation sequence is concealed using SurveyMonkey ® software based on a commercial, but unknown algorithm without a pre-identified sequence.

Blinding
Participants will know that the trials are testing different formats to present health recommendations but will be blinded with respect to what formats are being compared and to the group to which they are allocated. Participants will not be aware of their random allocation to the PLR or SLV format until disclosure. Thus, participants will be blinded for all outcomes except the secondary outcome of preference. We will blind data analysts involved in data interpretation and manuscript writing to reduce bias in data analysis and interpretation [25,26]. To reduce bias, we will draft each population's manuscript with general group labels (i.e., group A vs B) and agree on the final interpretation before the group allocation (optimized PLR or SLV) is revealed.

Sample size calculation
The sample size was calculated using the primary outcome of understanding. For this two-sided (α = 0.05) superiority analysis, these computations were made based on a t-test with the null hypothesis that there is no difference between the PLR version and the SLV in understanding of information. We will determine whether we can reject the null hypothesis of no difference in understanding between the formats of presenting recommendations. We assume a correct response rate (regarding the understanding of information) of 75% in the PLR arm and 60% in the control arm [relative risk =1.25] based on a recent RCT and prior trials on plain language summaries (PLS) of summary of findings table [19,23]. With typical alpha (0.05) and beta (0.8) parameters and an allocation ratio of 1:1, 240 participants (120 in each arm) would be required (Stata/SE 16.0) [27]. Assuming 15% non-completion, we will recruit 282 participants from each population and adjust this number if non-completion is higher.
Since we are using two different recommendation topics with each population, we might observe an interaction between the recommendation topic and our outcome of understanding. Thus, when we reach half of our intended sample size, we will conduct an interim analysis for a possible interaction effect. If the data suggests that there might be an interaction, we will consider modifying our initial sample size. This new sample size cannot be determined in advance since we cannot anticipate the magnitude of the potential interaction effect, but it will be determined based on published guidance on power calculations for credible subgroup effects [28]. If needed, we will submit an amendment that explains the new sample size and need for additional recruitment based on the magnitude of the interaction effect identified. In summary, an enhanced sample size, if indicated and feasible, will allow us to conduct a more meaningful analysis by exploring the results by topic of recommendation.

Consultation and pilot testing
We will use the same survey template for all populations; however, the "understanding" questions will be appropriately different for each recommendation. We pilot tested the surveys during the pre-trial engagement work to gather feedback from Canadian youth advisors, Cochrane Consumer Network, adult stakeholders, and parent stakeholders. We had 10 youth, 29 adults, and 5 parents pilot test the survey and provide feedback on the time to complete the survey, length of the survey, clarity of questions, Internet difficulties, and any other comments to enhance the survey experience. Revisions were made to the surveys using pilot test feedback until no errors or inconsistencies were detected and the surveys were easily understood. The survey language used for each population is tailored based on the feedback from stakeholders. Pilot test participant results will not be included in the final analysis.

Statistical analysis
Only participants who submit a complete good quality survey will be included in the analysis. We will exclude all incomplete, speeding (less than 6 min to completion), and straight-lining responses. Each population's results will be analyzed separately, and we will explore subgroup effects within each group. These subgroups are the topic of recommendation, health literacy level, and English proficiency level of participants. We will also pool the results across populations and conduct a meta-analysis for our primary outcome of "understanding" and other secondary outcomes. In the final analysis, after pooling the results, we will explore for potential interaction related to the recommendation topic and participants' health literacy level. We will conduct the analyses using IBM SPSS ® (Statistical Package for Social Sciences) version 23.

Descriptive analysis
We will summarize participant baseline characteristics and outcomes using means and standard deviations (SD) for continuous variables and proportions for categorical variables.

Inferential analysis
We will perform a primary analysis including all randomized participants. We will exclude participants who take less than 6 min to complete the survey. For the outcome of understanding, we will use χ 2 tests and risk difference with 95% CIs to compare the proportion of correct responses between groups. For the outcomes of accessibility and satisfaction, we will use t-tests and mean differences 95% confidence intervals (95% CIs) to compare means and SDs between the intervention and control groups. Finally, we will present preference as mean (SD) overall and for both trial arms. Skewness, Shapiro-Wilk tests, and histograms will be used to evaluate whether the distribution was shifted toward the same preference in both groups. Levene's test of equal variances will be used for all t-tests and degrees of freedom will be adjusted for p < 0.05. We will report all p-values to three decimal places, with values less than 0.001 reported as < 0.001. We will consider statistical significance at p < 0.05. Furthermore, we will conduct a meta-analysis across the three trials by combining results of the three trials. Our a priori hypothesis is that there is no difference across the three populations despite the differences in population characteristics and slight difference in the presentation formats for each population. However, we will explore differences between trials applying the GRADE approach to evaluating inconsistency which includes the chi-square test, the I 2 value, the overlap in confidence intervals, and the differences in the point estimates of effect for all outcomes [29].

Qualitative methods
We will interview a sample of survey participants to explore their user experience with two recommendation formats. These semi-structured interviews will help us contextualize the results of the survey and understand participants' preference for receiving the information.

Selection of participants
At the end of each survey, participants will be invited to contribute to a virtual, one-on-one, semi-structured interview, conducted by the research coordinator for each leading site. Interested participants will voluntarily provide their email address at the end of the survey, and the research coordinator will follow up with them through email.

Interviews
We will invite individuals from the list of participants who have agreed to participate in the qualitative interviews. For the parents' population, we will use purposive sampling (inviting participants based on region, ethnicity, gender, and topic of recommendation). For the adult and youth populations, we will not use a purposive sample; however, we will collect demographic data from the participants at the time of the interview. All interview participants will be compensated for their time with a gift card valued at $25 CAD.
Prior to the interview, participants will be provided with the consent form for the semi-structured interviews. Following that, a Zoom invite will be shared with the participant. Prior to the start of the interview, the research coordinator conducting the interview will obtain verbal consent from the participant for participating in the interview and for the interview to be audio-recorded. If the participant chooses to leave their video camera on during the interview, consent will be acquired for videorecording the interview. Consent will be documented on a log by the coordinator. If participants decline recording of the interviews, we will not move forward with the interview. We will transcribe interview recordings verbatim and de-identify all interview transcripts.
Interviews will be conducted in English by trained interviewers and will take approximately 30-60 min. Interviews will be guided by open-ended questions that cover the seven facets of the honeycomb model to verify user's experiences with the different recommendation formats: usefulness, usability, findability, accessibility, desirability, credibility, and value [30]. We will pilot test the interview guide within the three populations to refine the questions and language based on the populations' unique needs prior to the interviews.

Data management
All recordings will be stored on a password-protected device and secure network and accessed only by members of the research team. The recordings will be kept for 5 years, at which point they will be destroyed. All transcribed recordings will be reviewed and verified by the interviewer or a second research team member prior to being de-identified. NVivo software (QSR, 2018) will be used for data coding and management.

Data analysis
Qualitative data analysis will follow a hybrid inductive/ deductive method [31]. This method allows for flexibility to utilize an established framework during analysis (deductive) as well as identify codes that emerge from the data (inductive). The 7 elements of the honeycomb model for user experience (Fig. 4) informed the creation of the interview guide and were adapted and used as the initial model for analysis (see Supplementary Material 2 for the interview guide) [30]. The seven elements of the model will guide coding with inductive coding of data that extends beyond the model. Data collection and analysis will occur iteratively [32]. The analysis will follow six steps: (1) data cleaning-comparing the transcript with the recording and doing the cleaning to ensure alignment, (2) generating initial codes using line-by-line coding, (3) searching for themes guided by the honeycomb model, (4) reviewing the themes and generating a "thematic map, " (5) defining and naming themes guided by the honeycomb model, and (6) writing the report. Data collection will occur until saturation is reached [33].
We will develop joint displays, explicitly merging the results from the three populations' data sets through a side-by-side comparison to assess how findings are aligned among the three data sets [34].

Rigor
Strategies for maintaining rigor [35] will be used such as utilize a detailed study log and audit trail to ensure transparency. A coding system will be utilized and refined throughout the iterative data collection and analysis stages and each site will have a primary coder and a secondary coder to code approximately 10% of the transcripts and compare to maintain intra-rater reliability [35]. We will use iterative member checking, reflecting back statements made during interviews to participants for clarity and understanding.
We will also use our trial survey data to triangulate our findings with survey responses. Our analysis team will reflect on and disclose our own digital experiences and consider this in our iterative analysis process.

Reporting
We will report our quantitative findings according to the Consolidated Standards of Reporting Trials (CON-SORT) reporting guidelines and qualitative findings following the Consolidated criteria for reporting qualitative research (COREQ) [36,37]. For the quantitative results, we will pool the results across the three populations and conduct a meta-analysis for our primary outcome of "understanding" and other secondary outcomes. We will explore results for each population separately as well as compare results across populations.
Finally, we will crossmatch and compare the results of the qualitative interviews with those of the quantitative surveys and try to better understand and contextualize the RCT findings. Based on the results, we will discuss possible changes to the PLR format.

Discussion
Guideline developers typically create recommendations for health professionals or health agencies [1]. However, particularly in the context of COVID-19 and any future pandemics or topics of high public interest, the general public must have access to health recommendations that they can understand and trust [38]. For example, public understanding and acceptance is required for the implementation and uptake of many public health recommendations such as vaccination, using facial masks, and social distancing. Furthermore, parents are making decisions for children who cannot make decisions for themselves, and they need trustworthy recommendations specific to this unique population (e.g., vaccine safety and efficacy for children). On the other hand, youth access health recommendations about COVID-19 and make independent decisions about implementing public health measures, like social distancing and face mask use [3].
It has become crucial to engage end-users in shaping behaviors and activities that will eventually affect them and their communities. Their involvement will improve the relevance, usefulness, and transparency of guidelines [39]. More importantly, it will ensure that these guideline products are understandable and acceptable. We believe that when the public is presented with plain language recommendations, they will have a better understanding of guideline recommendations. These trials are designed with the input of end-users from three populations: youth, adults, and parents. Results will inform the best ways to make recommendations understandable and accessible to the public, thus increasing the public's confidence in science and evidence uptake.